{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "description: \"Logging... logging everywhere! \\U0001F52E\"\n",
    "---\n",
    "\n",
    "# Loggers\n",
    "\n",
    "Welcome to the _\"Logging\"_ tutorial of the _\"From Zero to Hero\"_ series. In this part we will present the functionalities offered by the _Avalanche_ `logging` module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "!pip install avalanche-lib==0.5"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 📑 The Logging Module\n",
    "\n",
    "In the previous tutorial we have learned how to evaluate a continual learning algorithm in _Avalanche_, through different metrics that can be used _off-the-shelf_ via the _Evaluation Plugin_ or stand-alone. However, computing metrics and collecting results, may not be enough at times.\n",
    "\n",
    "While running complex experiments with long _waiting times_, **logging** results over-time is fundamental to \"_babysit_\" your experiments in real-time, or even understand what went wrong in the aftermath.\n",
    "\n",
    "This is why in Avalanche we decided to put a strong emphasis on logging and **provide a number of loggers** that can be used with any set of metrics!\n",
    "\n",
    "### Loggers\n",
    "\n",
    "_Avalanche_ at the moment supports four main Loggers:\n",
    "\n",
    "* **InteractiveLogger**: This logger provides a nice progress bar and displays real-time metrics results in an interactive way \\(meant for `stdout`\\).\n",
    "* **TextLogger**: This logger, mostly intended for file logging, is the plain text version of the `InteractiveLogger`. Keep in mind that it may be very verbose.\n",
    "* **TensorboardLogger**: It logs all the metrics on [Tensorboard](https://www.tensorflow.org/tensorboard) in real-time. Perfect for real-time plotting.\n",
    "* **WandBLogger**: It leverages [Weights and Biases](https://wandb.ai/site) tools to log metrics and results on a dashboard. It requires a W&B account.\n",
    "\n",
    "In order to keep track of when each metric value has been logged, we leverage two `global counters`, one for the training phase, one for the evaluation phase. \n",
    "You can see the `global counter` value reported in the x axis of the logged plots.\n",
    "\n",
    "Each `global counter` is an ever-increasing value which starts from 0 and it is increased by one each time a training/evaluation iteration is performed (i.e. after each training/evaluation minibatch).\n",
    "The `global counters` are updated automatically by the strategy.\n",
    "\n",
    "#### How to use loggers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from torch.optim import SGD\n",
    "from torch.nn import CrossEntropyLoss\n",
    "from avalanche.benchmarks.classic import SplitMNIST\n",
    "from avalanche.evaluation.metrics import forgetting_metrics, \\\n",
    "accuracy_metrics, loss_metrics, timing_metrics, cpu_usage_metrics, \\\n",
    "confusion_matrix_metrics, disk_usage_metrics\n",
    "from avalanche.models import SimpleMLP\n",
    "from avalanche.logging import InteractiveLogger, TextLogger, TensorboardLogger, WandBLogger\n",
    "from avalanche.training.plugins import EvaluationPlugin\n",
    "from avalanche.training import Naive\n",
    "\n",
    "benchmark = SplitMNIST(n_experiences=5, return_task_id=False)\n",
    "\n",
    "# MODEL CREATION\n",
    "model = SimpleMLP(num_classes=benchmark.n_classes)\n",
    "\n",
    "# DEFINE THE EVALUATION PLUGIN and LOGGERS\n",
    "# The evaluation plugin manages the metrics computation.\n",
    "# It takes as argument a list of metrics, collectes their results and returns\n",
    "# them to the strategy it is attached to.\n",
    "\n",
    "\n",
    "loggers = []\n",
    "\n",
    "# log to Tensorboard\n",
    "loggers.append(TensorboardLogger())\n",
    "\n",
    "# log to text file\n",
    "loggers.append(TextLogger(open('log.txt', 'a')))\n",
    "\n",
    "# print to stdout\n",
    "loggers.append(InteractiveLogger())\n",
    "\n",
    "# W&B logger - comment this if you don't have a W&B account\n",
    "loggers.append(WandBLogger(project_name=\"avalanche\", run_name=\"test\"))\n",
    "\n",
    "eval_plugin = EvaluationPlugin(\n",
    "    accuracy_metrics(minibatch=True, epoch=True, experience=True, stream=True),\n",
    "    loss_metrics(minibatch=True, epoch=True, experience=True, stream=True),\n",
    "    timing_metrics(epoch=True, epoch_running=True),\n",
    "    cpu_usage_metrics(experience=True),\n",
    "    forgetting_metrics(experience=True, stream=True),\n",
    "    confusion_matrix_metrics(num_classes=benchmark.n_classes, save_image=True,\n",
    "                             stream=True),\n",
    "    disk_usage_metrics(minibatch=True, epoch=True, experience=True, stream=True),\n",
    "    loggers=loggers\n",
    ")\n",
    "\n",
    "# CREATE THE STRATEGY INSTANCE (NAIVE)\n",
    "cl_strategy = Naive(\n",
    "    model, SGD(model.parameters(), lr=0.001, momentum=0.9),\n",
    "    CrossEntropyLoss(), train_mb_size=500, train_epochs=1, eval_mb_size=100,\n",
    "    evaluator=eval_plugin)\n",
    "\n",
    "# TRAINING LOOP\n",
    "print('Starting experiment...')\n",
    "results = []\n",
    "for experience in benchmark.train_stream:\n",
    "    # train returns a dictionary which contains all the metric values\n",
    "    res = cl_strategy.train(experience)\n",
    "    print('Training completed')\n",
    "\n",
    "    print('Computing accuracy on the whole test set')\n",
    "    # test also returns a dictionary which contains all the metric values\n",
    "    results.append(cl_strategy.eval(benchmark.test_stream))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# need to manually call W&B run end since we are in a notebook\n",
    "import wandb\n",
    "wandb.finish()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext tensorboard"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%tensorboard --logdir tb_data --port 6066"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create your Logger\n",
    "\n",
    "If the available loggers are not sufficient to suit your needs, you can always implement a custom logger by specializing the behaviors of the `StrategyLogger` base class.\n",
    "\n",
    "This completes the \"_Logging_\" tutorial for the \"_From Zero to Hero_\" series. We hope you enjoyed it!"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 🤝 Run it on Google Colab\n",
    "\n",
    "You can run _this chapter_ and play with it on Google Colaboratory: [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ContinualAI/avalanche/blob/master/notebooks/from-zero-to-hero-tutorial/06_loggers.ipynb)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
